Collaborative Cluster Configuration for Distributed Data-Parallel Processing: A Research Overview

Many organizations routinely analyze large datasets using systems for distributed data-parallel processing and clusters of commodity resources. Yet, users need to configure adequate resources their data jobs. This requires significant insights into expected job runtimes scaling behavior, resource characteristics, input distributions, other factors. Unable estimate performance accurately, frequently overprovision jobs, leading low utilization high costs. In this paper, we present major building blocks towards a collaborative approach optimization cluster configurations based on runtime models. We believe that can be shared used models across different execution contexts, significantly reducing the reliance recurrence individual jobs or, else, dedicated profiling. For this, describe how similarity infrastructures employed combine suitable points from local global executions accurate Furthermore, outline approaches prediction via more context-aware reusable Finally, lay out metrics previous combined with monitoring effectively re-configure dynamically.